Profiling Distributed File Systems with Computer Animation

Achieving performance, reliability, and scalability has proven difficult for distributed file systems. Placement of data, load distribution and other overheads are often the culprits. Profiling is a useful technique for understanding file system behavior, improving performance and debugging problems. Existing file system profiling methods often examine fine-grained system activity, such as the path of a single request. This makes understanding interactive relationships difficult. Also, current profiling techniques do not provide a way for users to easily view full system behavior, often relying on logs, statistics, or simple graphs. We present a distributed file system profiling method based on clear-box profiling and visualization of full system behavior. Our approach allows for portable, low-overhead profiling and provides users with real-time animation of system behavior. We present users with a visualization of the file system architecture and animate behaviors, such as load distribution, data location, and network traffic, under real workloads. This allows users to visually identify bottlenecks, latencies, characterize system state and I/O requests, and debug problems. We evaluate the overhead and effectiveness of our profiling method on the Ceph petabyte-scale, parallel file system. Our evaluation shows the visualization client is able to profile each device with very minimal overhead. Additionally, We demonstrate the usefulness of our profiling and visualization techniques through a series of profiles on Ceph. Using these profiles, we have discovered several interesting and important problems ranging from inefficiencies, namely in small I/O operations, to major issues, such as instabilities in the Ceph messaging layer.